Encoding and decoding machine with recurrent neural networks

ABSTRACT

Techniques for reconstructing a signal encoded with a time encoding machine (TEM) using a recurrent neural network including receiving a TEM-encoded signal, processing the TEM-encoded signal, and reconstructing the TEM-encoded signal with a recurrent neural network.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of International Application SerialNo. PCT/US2012/024413, filed Feb. 9, 2012 which claims priority to U.S.Provisional Application Ser. No. 61/441,203, filed Feb. 9, 2011, each ofwhich is hereby incorporated by reference in it's entirety.

STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT

This invention was made with government support under grantsFA9550-01-1-0350, awarded by USAR/AFOSR, CNS 0855217, awarded by theNational Science Foundation, and CNS 0958379, awarded by the NationalScience Foundation. The government has certain rights in the invention.

BACKGROUND

The presently disclosed subject matter relates to methods and systemsfor the encoding and reconstruction of signals encoded with timeencoding machines (TEM), and more particularly to the reconstruction ofsignals encoded with TEMs with the use of recurrent neural networks.

Most signals in the natural world are analog, i.e., they cover acontinuous range of amplitude values. However, most computer systems forprocessing these signals are binary digital systems. Synchronousanalog-to-digital (A/D) converters can be used to capture analog signalsand present a digital approximation of the input signal to a computerprocessor. That is, at certain moments in time synchronized to a systemclock, the amplitude of the signal of interest is captured as a digitalvalue. When sampling the amplitude of an analog signal, each bit in thedigital representation of the signal represents an increment of voltage,which defines the resolution of the A/D converter. Analog-to-digitalconversion is used in many applications, such as communications where asignal to be communicated can be converted from an analog signal, suchas voice, to a digital signal prior to transport along a transmissionline.

Applying traditional sampling theory, a band limited signal can berepresented with a quantifiable error by sampling the analog signal at asampling rate at or above what is commonly referred to as the Nyquistsampling rate. It is a trend in electronic circuit design to reduce theavailable operating voltage provided to integrated circuit devices. Inthis regard, power supply voltages for circuits are generallydecreasing. While digital signals can be processed at the lower supplyvoltages, traditional synchronous sampling of the amplitude of a signalbecomes more difficult as the available power supply voltage is reducedand each bit in the A/D or D/A converter reflects a substantially lowervoltage increment.

Time Encoding Machines (TEMs) can encode analog information in the timedomain using only asynchronous circuits. Representation in the timedomain can be an alternative to the classical sampling representation inthe amplitude domain. Applications for TEMs can be found in low powernano-sensors for analog-to-discrete (A/D) conversion as well as inmodeling olfactory systems, vision and audition in neuroscience.

SUMMARY

Methods and systems for reconstructing TEM-encoded signals usingrecurrent neural networks are disclosed herein.

According to some embodiments of the disclosed subject matter, methodsfor reconstructing a signal encoded with a time encoding machine (TEM)using a recurrent neural network include first receiving a TEM-encodedsignal and processing it for input into a recurrent neural network. Theoriginal signal is then reconstructed using the recurrent neuralnetwork.

In one embodiment, the reconstruction process includes formulating thereconstruction into a variational problem having a solution equal to asummation of a series of functions multiplied by a series ofcoefficients. The coefficients can be obtain by solving an optimizationproblem. The optimization problem can be solved by a recurrent neuralnetwork whose architecture can be defined according to a particulardifferential equation. The original analog signal can then bereconstructed using the coefficients.

In another embodiment, the TEM-encoded signal can be Video Time EncodingMachine (vTEM) encoded signals. The method of reconstructing using arecurrent neural network can have a plurality of inputs for a pluralityof signals generated by the vTEM.

According to some embodiments of the disclosed subject matter, systemsfor reconstructing a signal encoded with a TEM using a recurrent neuralnetwork include at least one input for receiving a TEM-encoded signal.The input can then pass a signal along an arrangement of adders,integrators, multipliers, and/or piecewise linear activators arranged toreflect a recurrent neural network defined by a particular differentialequation. The system can have a plurality of outputs for providing atleast one coefficient representing a reconstructed signal.

In some embodiments, the recurrent neural network can include threelayers. The first layer can consist of a plurality of multiply/addunits. Each of the nodes in the first layer can be operatively connectedto a second layer which also includes a plurality of multiply/add units.The third layer, also consisting of a plurality of multiply/add unitscan calculate a gradient weighted by a learning rate and can output thetime derivative of the coefficient. The outputs can then be integratedand fed back into the first layer.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic diagram of a recurrent neural network forreconstructing a TEM-encoded signal according to one embodiment of thedisclosed subject matter.

FIG. 2A-FIG. 2B is a schematic diagram of a recurrent neural network forreconstructing a TEM-encoded signal according to another embodiment ofthe disclosed subject matter.

FIG. 3 is a schematic diagram of a Video Time Encoding Machine (vTEM).

FIG. 4 is a schematic diagram of a recurrent neural network according toone embodiment the disclosed subject matter for reconstructing vTEMencoded signals.

FIG. 5A-FIG. 5B is schematic diagram of a recurrent neural networkaccording to another embodiment the disclosed subject matter forreconstructing vTEM encoded signals.

FIG. 6. illustrates how a signal can be divided into overlapping volumesand stitched according to one embodiment of the disclosed subjectmatter.

DETAILED DESCRIPTION

The presently disclosed subject matter provides techniques for encodingand decoding an analog signal into the time domain, also referred to asthe spike domain. More particularly, the presently disclosed subjectmatter provides for the use of recurrent neural networks in encoding anddecoding analog signals into and from the time domain and the encodingand decoding of visual stimuli with recurrent neural networks.

The present application makes use of time encoding, a time encodingmachine (TEM) and a time decoding machine (TDM). Time encoding is areal-time asynchronous mechanism of mapping the amplitude of abandlimited signal u=u(t), tε

, into a strictly increasing time sequence (t_(k)), kε

, where

and

denote the sets of real numbers and integers, respectively. It should benoted that throughout this specification the symbol

should be construed in the same manner as the symbol

, to mean the set of real numbers A Time Encoding Machine (TEM) is therealization of an asynchronous time encoding mechanism. A Time DecodingMachine (TDM) is the realization of an algorithm for signal recovery.With increasing device speeds TEMs are able to better leverage atemporal model of encoding a signal. The interest in temporal encodingin neuroscience is closely linked with the natural representation ofsensory stimuli (signals) as a sequence of action potentials (spikes).Spikes can be discrete time events that carry information about stimuli.

Time Encoding Machines (TEMs) model the representation (encoding) ofstimuli by sensory systems with neural circuits that communicate viaspikes (action potentials). TEMs asynchronously encode time-varyinganalog stimuli into a multidimensional time sequence. TEMs can also beimplemented in hardware. For example Asynchronous Sigma-Delta Modulators(ASDMs), that have been shown to be an instance of TEMs, can be robustlyimplemented in low power analog VLSI. With the ever decreasing voltageand increasing clock rate, amplitude domain high precision quantizersare more and more difficult to implement. Representing information intime domain follows the miniaturization trends of nanotechnology anddemonstrates its potential as next generation silicon based signalencoders.

Asynchronous Sigma/Delta modulators as well as FM modulators can encodeinformation in the time domain as described in “Perfect Recovery andSensitivity Analysis of Time Encoded Bandlimited Signals” by A. A. Lazarand L. T. Toth (IEEE Transactions on Circuits and Systems-I: RegularPapers, 51(10):2060-2073, October 2004), which is incorporated byreference. More general TEMs with multiplicative coupling, feedforwardand feedback have also been characterized by A. A. Lazar in “TimeEncoding Machines with Multiplicative Coupling, Feedback andFeedforward” (IEEE Transactions on Circuits and Systems II: ExpressBriefs, 53(8):672-676, August 2006), which is incorporated by reference.TEMs realized as single and as a population of integrate-and-fireneurons are described by A. A. Lazar in “Multichannel Time Encoding withIntegrate-and-Fire Neurons” (Neurocomputing, 65-66:401-407, 2005) and“Information Representation with an Ensemble of Hodgkin-Huxley Neurons”(Neurocomputing, 70:1764-1771, June 2007), both of which areincorporated by reference. Single-input multiple-output (SIMO) TEMs aredescribed in “Faithful Representation of Stimuli with a Population ofIntegrate-and-Fire Neurons” by A. A. Lazar and E. A. Pnevmatikakis(Neural Computation), which is incorporated by reference.

Disclosed herein are methods and systems of reconstructing a signalencoded with a time encoding machine (TEM) using a recurrent neuralnetwork. Examples will now be given showing exemplary embodiments of thedisclosed subject matter. First, a method and system are provided forreconstruction of a TEM-encoded signal encoded with a single-inputsingle-output TEM. For purposes of illustration and not limitation, amodel of the signal and an overview of the encoding process areprovided. One of ordinary skill will appreciate that other suitablemodels and encoding processes can be used in accordance with the subjectmatter disclosed herein. Next, the methods and systems forreconstruction of a single-input single-output TEM-encoded signal willbe expanded to reconstruction of multi-dimensional signals, for examplethe reconstruction of space-time stimuli encoded with Video TimeEncoding Machines (vTEMs). For purposes of illustration, the exemplarymodel of the single-input single-output signal is extended to thespace-time vTEM encoded signal. An overview of vTEM encoding will alsobe provided. One of ordinary skill will appreciate that a variety ofother suitable models and encoding processes can be used, and that theexamples provided herein are not for purposes of limitation. Forexample, the presently disclosed subject matter can reconstructolfactory or auditory stimuli as well as visual stimuli.

Single-Input Single-Output Tem Encoded Signals

In the case of a single-input single-output TEM that encodestime-varying signals, time-varying stimuli (signals) can be elements ofa space of trigonometric polynomials. For example, in Hilbert space

, every element u=u(t), tε

is of the form

${{u(t)} = {\sum\limits_{m_{t} = {- M_{t}}}^{M_{t}}{c_{m_{t}}e_{m_{t}}}}},$

with

$e_{m_{t}} = {\exp ( {j\; m_{t}\frac{\Omega_{t}}{M_{t}}t} )}$

an element of the basis spanning the space

_(t); Ω_(t) and M_(t) are the bandwidth and the order of the space oftrigonometric polynomials, respectively. Every element in this Hilbertspace is periodic with period

$S_{t} = {\frac{2\pi \; M_{t}}{\Omega_{t}}.}$

Assuming that all signals are real, c_(−m) _(t) = c_(m) _(t) , where (·)denotes the complex conjugate.

The inner product in

_(t) can be defined in the usual way: ∀u, vε

_(t),

${\langle{u,v}\rangle} = {\frac{1}{S_{t}}{\int_{0}^{S_{t}}{{u(t)}\overset{\_}{v(t)}{{t}.}}}}$

The space of trigonometric polynomials can be a finite dimensionalHilbert space, and therefore, a Reproducing Kernel Hilbert Space (RKHS),with reproducing kernel

${{K( {t,s} )} = {\sum\limits_{m_{t} = {- M_{t}}}^{M_{t}}{e_{m_{t}}( {t - s} )}}},$

with t, sε

.

Modeling the set of stimuli in a Hilbert space can enable the use ofgeometry of the space to reduce stimulus encoding to projections on aset of functions.

Encoding of a time-varying signal, for example, with a TEM, can consistof two cascaded modules. The signal can be passed through a temporalreceptive field D_(T)(t) before being fed into a neural circuit. Theprocessing of the temporal receptive field can be modeled as filtering.For example, an operator ^(r)L:

_(t)→

_(t), can be defined such that

v(t)=^(r) Lu=

D _(T)(t−s)u(s)ds=(D _(T) *u)(t).

The neural circuit can encode the output of the receptive field. Theneural circuit can be realized with different neural models. Forexample, the neural circuit can be an Integrate-And-Fire (IAF) neuron, aHodgkin-Huxley neuron, an Asynchronous Sigma-Delta Modulator (ASDM), orother suitable model. Spike times of a spike train at the output of theneural circuit can be modeled as (t_(k)), k=0, 1, 2, . . . , n.

The operation of the neural circuit can be described by a bounded linearfunctional ^(T)L_(k):

_(t)→

. The explicit formula of this functional can be determined by thet-transform of the neuron, given by ^(T)L_(k)u=q_(k), for u=ε

_(t), where ^(T)L_(k) and q_(k) usually depend on (t_(k)), k=0, 1, 2, .. . , n. For example, for an ideal IAF neuron with the t-transform givenby ∫_(k) ^(k+1)u(s)ds=κδ−b(t_(k+1)−t_(k)), leads to ^(T)L_(k)u=∫_(k)^(k+1)u(s)ds, q_(k)=κδ−b(t_(k+1)−t_(k)), where κ, δ and b are,respectively, the integration constant, the threshold and the bias ofthe IAF neuron.

Combining the two cascaded modules together, bounded linear functionalsL_(k):

_(t)→

can be defined as L_(k)=^(T)L_(k) ^(r)L so that L_(k)u=^(T)L_(k)^(r)Lu=q_(k). By the Riesz representation theorem, these functionals canbe expressed in inner product form as L_(k)u=

u,φ_(k)

,for all uε

where, by the reproducing property, φ_(k)(t)=

φ_(k), K_(t)

=L_(k) K_(t) , with K_(t)(s)=K(t,s).

Because the inner products are merely projections of the time-varyingstimulus onto the axes defined by the φ_(k)'s, encoding can beinterpreted as generalized sampling, and the q_(k)'s are measurementsgiven by sampling the signal. Unlike traditional sampling, the samplingfunctions in time encoding are signal dependent.

In one aspect of the disclosed subject matter, signals encoded with atime encoding machine (TEM) can be reconstructed using a recurrentneural network. In some embodiments, the recurrent neural network candecode a TEM-encoded signal that has been encoded by a single-inputsingle-output TEM as described above. Reference will now be made toparticular embodiments of the disclosed subject matter forreconstructing a TEM-encoded signal with a recurrent neural network forpurposes of illustration. However, one of ordinary skill will recognizethat other suitable variations exist, and thus the following discussionis not intended to be limiting.

Reconstruction of a time-varying signal u from a TEM-encoded signal canbe formulated as a variational problem. In one embodiment, thereconstruction is formulated into the variational problem

${\hat{u} = {\underset{u \in \mathcal{H}}{argmin}\{ {{\sum\limits_{k = 1}^{n}( {{\langle{u,\varphi_{k}}\rangle} - q_{k}} )^{2}} + {n\; \lambda {u}_{\mathcal{H}}^{2}}} \}}},$

where λ is a smoothing parameter. By the Representer Theorem, thesolution to this problem takes the form

$\hat{u} = {\sum\limits_{k = 1}^{n}{c_{k}{\varphi_{k}.}}}$

Substituting the solution into the problem, the coefficients c_(k) canbe obtained by solving the unconstrained optimization problem minimize∥Gc−q∥_(1/2) ²+nλc^(T)Gc, where c=[c₁, c₂, . . . , c_(n)]^(T), q=[q₁,q₂, . . . . , q_(n)]^(T), I is the n×n identity matrix and G is asymmetric matrix with entries

$\lbrack G\rbrack_{k,l} = {{\langle{\varphi_{k},\varphi_{l}}\rangle} = {\sum\limits_{m_{t} = {- M_{t}}}^{M_{t}}{( {\int_{k}^{k + 1}{( {D_{T}*e_{- m_{i}}} )(s){{s} \cdot {\int_{l}^{l + 1}{( {D_{T}*e_{m_{t}}} )(s){s}}}}}} ).}}}$

This minimization problem has an explicit analytical solution with c thesolution of the system of linear equations G^(T)(G+nλI)c=G^(T)q.Therefore, reconstruction of a time-varying signal can be accomplishedby solving a system of linear equations.

This system of linear equations can be solved by, particularly in thecase where the matrix G is a singular matrix, taking the Moore-Penrosepseudo-inverse (hereinafter, “pseudo-inverse”). However, the calculatingthe Moore-Pensore pseudo-inverse is typically computationally intensive.For example, one conventional algorithm for evaluating thepseudo-inverse is based on singular value decomposition (SVD). SVD isparticularly computationally demanding. Recurrent neural networks can beused to efficiently solve optimization problems. These networks can havestructures that can be easily implemented in analog VLSI.

In one embodiment, a recurrent neural network can be used to solve thesystem of linear equations. For example, using a general gradientapproach for solving the unconstrained optimization problemminimize∥Gc−q∥_(1/2) ²+nλc^(T)Gc, a set of differential equations can beconsidered:

${\frac{c}{t} = {{- \mu}{\nabla{E(c)}}}},$

with initial condition c(0)=0, where E(c)=½(∥Gc−q∥_(1/2) ²+nλc^(T)Gc),and μ(c, t) is a n×n symmetric positive definite matrix that determinesthe speed of convergence and whose entries are usually dependent on thevariables c(t) and time t, define the architecture of the recurrentneural network. It follows that ∇E(c)=G^(T)((G+nλI)c−q). Because E(c) isconvex in c, the system of differential equations asymptoticallyapproaches the unique solution of the regularized optimization problemminimize∥Gc−q∥_(1/2) ²+nλc^(T)Gc. Consequently,

$\frac{c}{t} = {{- \mu}\; {{G^{T}( {{( {G + {n\; \lambda \; I}} )c} - q} )}.}}$

The set of differential equations can be mapped into a recurrent neuralnetwork, for example as depicted in FIG. 1. In one embodiment, thenetwork can be a three layer neural network. In the first layer 110,consisting of a plurality (n) of multiply/add units, the vector(G+nλJ)c−q can be computed. The multiplication factors can be theentries of the matrix G+nλI and the vector q. In the second layer 120,∇E(c) can be evaluated. The second layer 120 can also comprise nmultiply/add units, with the multiplication factors provided by theentries of the matrix G. In the third layer 130, the gradient isweighted by the learning rate p. The third layer 130 can also consist ofn multiply/add units 140 and 145. The outputs of the third layer canprovide the time derivative of the vector c(t). The time derivatives canthen be integrated with integrator 150, providing the outputs 160 of therecurrent neural network, and the outputs can be fed back to the firstlayer.

Alternatively, in another embodiment, the reconstruction of theTEM-encoded signal can be formulated as the spline interpolation problem

${\hat{u} = {\underset{{u \in \mathcal{H}},{\{{{\mathcal{L}_{}u} = q_{k}}\}}_{k = 1}^{n}}{argmin}\{ {u}_{\mathcal{H}}^{2} \}}},$

that seeks to minimize the norm as well as satisfy all the t-transformequations. This problem can also have a solution that takes the form

$\hat{u} = {\sum\limits_{k = 1}^{n}{c_{k}{\varphi_{k}.}}}$

Substituting the solution into the interpolation problem, the vector ofcoefficients c are the solution of the optimization problem

${minimize}\mspace{14mu} \frac{1}{2}c^{T}{Gc}$subject  to  Gc = q,

where c=[c₁, c₂, . . . , c_(n)]^(T), q=[q₁, q₂, . . . , q_(n)]^(T), andG is a symmetric matrix with entries

$\lbrack G\rbrack_{k,l} = {{\langle{\varphi_{k},\varphi_{l}}\rangle} = {\sum\limits_{m_{t} = {- M_{t}}}^{M_{t}}{( {\int_{k}^{k + 1}{( {D_{T}*e_{- m}} )(s){{s} \cdot {\int_{l}^{l + 1}{( {D_{T}*c_{m}} )(s){s}}}}}} ).}}}$

Due to the RKHS property, G is a positive semidefinite matrix.Therefore, the optimization problem is a convex quadratic programmingproblem with equality constraints.

The optimization problem can be reformulated as a standard quadraticprogramming problem. By setting x=[x₊ ^(T)x⁻ ^(T)]^(T) and imposingx₊≧0, x⁻≧0 such that c=x₊−x⁻, the convex programming problem

$\begin{matrix}{{minimize}\mspace{14mu} \frac{1}{2}x^{T}{Qx}} \\{{{{subject}\mspace{14mu} {to}\mspace{14mu} {Ax}} = q},{x \geq 0}}\end{matrix}$

is obtained, where

${Q = \begin{bmatrix}G & {- G} \\{- G} & G\end{bmatrix}},$

and A=[G−G].

A recurrent network can be constructed that solves the convexprogramming problem, given by the differential equation

${{\frac{}{t}\begin{pmatrix}x \\y\end{pmatrix}} = {\beta \begin{pmatrix}{( {x - {\alpha \; Q\; x} + {\alpha \; A^{T}y}} )^{+} - x} \\{\alpha ( {{- {Ax}} + q} )}\end{pmatrix}}},$

where (x)⁺=[(x₁)⁺, . . . , (x_(n))⁺]^(T) and (x_(i))⁺=max{0, x_(i)}, αis a positive constant and β>0 is the scaling constant. In oneembodiment, this network can be a neural network depicted in FIG.2A-FIG. 2B, where β=1. It should be noted thatQx=[(Gc)^(T)−(Gc)^(T)]^(T), A^(T)y=[(Gy)^(T)−(Gy)^(T)]^(T) and Ax=Gc,allowing for the circuit diagram in FIG. 2A-FIG. 2B to be simplified.

In various embodiments, the recurrent neural networks, including thosedescribed above, can be realized with adders, integrators, multipliersand piecewise linear activation functions. The recurrent neural networksdisclosed herein can be highly parallel, and thus can solve large scaleproblems in real-time implemented in analog VLSI.

vTEM Encoded Signals

Just like in the case of single-dimension single-input single-outTEM-encoded signals, visual signals can be modeled as elements of thevector space of tri-variable trigonometric polynomials, denoted by

. Each element Iε

is of the form

${{I( {x,y,t} )} = {\sum\limits_{m_{x} = {- M_{x}}}^{M_{x}}{\sum\limits_{m_{y} = {- M_{y}}}^{M_{y}}{\sum\limits_{m_{t} = {- M_{t}}}^{M_{t}}{c_{m_{x},m_{y},m_{t}}e_{m_{x},m_{y},m_{t}}}}}}},$

where c_(m) _(x) _(,m) _(y) _(,m) _(t) ε

are constants and

${{e_{m_{x},m_{y},m_{t}}( {x,y,t} )} = {{{e_{m_{x}}(x)}{e_{m_{y}}(y)}{e_{m_{t}}(t)}} = {\exp ( {{j\; m_{x}\frac{\Omega_{x}}{M_{x}}x} + {j\; m_{y}\frac{\Omega_{y}}{M_{y}}y} + {j\; m_{t}\frac{\Omega_{t}}{M_{t}}}} )}}},{m_{x} = {- M_{x}}},\ldots \mspace{14mu},M_{x},$

m_(y)=−M_(y), . . . , M_(y), m_(t)=−M_(t), . . . , M_(t), constitute abasis of

and (x, y, t)ε

³. (Ω_(x), Ω_(y), Ω_(t)) and (M_(x), M_(y), M_(t)) are, respectively,the bandwidth and the order of the trigonometric polynomials in eachvariable. An element Iε

is also, respectively, periodic in each variable with period

${S_{x} = \frac{2\pi \; M_{x}}{~\Omega_{x}}},{S_{y} = \frac{2\pi \; M_{y}}{\Omega_{y}}},{S_{i} = {\frac{2\pi \; M_{t}}{\Omega_{t}}.}}$

The inner product can be defined ∀I₁I₂ε

as,

${\langle{I_{1},I_{2}}\rangle} = {\frac{1}{S_{x}S_{y}S_{t}}{\int_{0}^{x}{\int_{0}^{y}{\int_{0}^{t}{{I_{1}( {x,y,t} )}{I_{2}( {x,y,t} )}{x}{y}{{t}.}}}}}}$

By defining the inner product as such, the space of trigonometricpolynomials is a Hilbert space. Since

is finite dimensional it is also a RKHS with reproducing kernel

${K( {x,y,{t;x^{\prime}},y^{\prime},t^{\prime}} )} = {\sum\limits_{m_{x} = {- M_{x}}}^{M_{x}}{\sum\limits_{m_{y} = {- M_{y}}}^{M_{y}}{\sum\limits_{m_{t} = {- M_{t}}}^{M_{t}}{{e_{m_{x},m_{y},m_{t}}( {{x - x^{\prime}},{y - y^{\prime}},{t - t^{\prime}}} )}.}}}}$

Video Time Encoding Machines (vTEMs) encode space-time signals into thespike domain. FIG. 3 depicts a diagram of a typical vTEM. A typical vTEMcan comprise a plurality of receptive fields 310 and a population ofneurons 320. For example, a vTEM can consist of a plurality of branches330, each consisting of two modules in cascade: a visual receptive field310 a and a neural circuit 320 a consisting of, for example an IAFneuron.

The receptive fields can be considered as spatio-temporal linear filtersthat preprocess the visual signals and feed them into the IAF neurons.The operation of the jth visual receptive field D^(j)(x,y,t) can begiven by the operator ^(S)L^(j):

→

_(t) by ^(s)L^(j)I=

(

D^(j)(x,y,s)I(x,y,t−s)dxdy)ds

_(t) denotes the univariable trigonometric polynomial space withbandwidth Ω_(t) and order M_(t). The operator maps a 3-D space into a1-D space.

A simplified case of the visual receptive field, where the field isspatio-temporally separable, can be considered. In this case, thereceptive fields can be separated into spatial receptive field D_(s)^(j)(x, y) and temporal receptive field D_(T) ^(j)(t) such that D^(J)(x,y, t)=D_(s) ^(j)(x,y)D_(T) ^(j)(t). The spatial receptive fields can be,for example, Gabor receptive fields or Difference of Gaussian receptivefields. Each spatial receptive field can be derived from a motherwavelet. For example, given the mother wavelet γ(x, y), the set of allreceptive fields can be obtained by performing the following threeoperations on their combinations: Dilation D_(α), αε

₊:

${{D_{\alpha}{\gamma ( {x,y} )}} = {{\alpha }^{- 1}{\gamma ( {\frac{x}{\alpha},\frac{y}{\alpha}} )}}},$

Rotation R_(θ), θε[0,2π):R_(θ)γ(x, y)=γ(x cos θ+y sin θ,−x sin θ+y cosθ), and Translation T_(x) ₀ _(y) ₀ , (x₀, y₀)ε

²:T_((x) ₀ _(,y) ₀ ₎γ(x, y)=γ(x−x₀, y−y₀). The receptive fields can forma from and cover the entire spatial field of a visual signal.

Each output of the visual receptive fields can then be fed into a neuralcircuit. The output of the jth neural circuit can be denoted (t_(k)^(j)), k=1, 2, . . . , n_(j), and the operation of the neural circuitcan be described by a bounded linear functional ^(T)L_(k) ^(j):

_(t)→

, where ^(T)L_(k) ^(j)=q_(k) ^(j); for uε

_(t).

Combining the two cascaded modules together and assuming a total of Nvisual receptive fields, the jth of which is connected to the jth neuralcircuit that generates one spike train (t_(k) ^(j)), k=1, 2, . . . ,n_(j), j=1, 2, . . . , N, bounded linear functionals ^(T)L_(k) ^(j):

_(t)→

can be defined as L_(k) ^(j)=^(T)L_(k) ^(jS)L^(j) so that L_(k)^(j)I=^(T)L_(k) ^(j)I=^(T)L_(k) ^(jS)L^(j)I=

I,φ_(k) ^(j)

=q_(k) ^(j), where φ_(k) ^(j)(x, y, t)=

φ_(k) ^(j), K_(x,y,t)

=L_(k) ^(j) K_(x,y,t) , with K_(x,y,t)(x′,y′,t′)=K(x,y,t;x′, y′,t′).Thus, as in the case of single-input single-output TEM, the operation ofthe vTEMs can be reduced to generalized sampling for functionals in

.

In one aspect of the disclosed subject matter, signals encoded with avideo time encoding machine (vTEM) can be reconstructed using arecurrent neural network. For purposes of illustration, reference willnow be made to particular embodiments of the disclosed subject matterfor reconstructing a vTEM-encoded signal with a recurrent neuralnetwork. However, one of ordinary skill will recognize that othersuitable variations exist, and thus the following discussion is notintended to be limiting.

As in the case of single-input single-output TEMs, the output striketrains of vTEMs can be used for reconstruction of the video signal. Inone embodiment, the notion of the single-input single-output case can beadapted and reconstruction can again be formulated as a variationalproblem of the form

${\hat{I} = {\underset{I\; {\varepsilon\mathcal{H}}}{argmin}\{ {{\sum\limits_{j = 1}^{}{\sum\limits_{k = 1}^{n_{j}}( {{\langle{I,\varphi_{k}^{j}}\rangle} - q_{k}^{j}} )^{2}}} + {n\; \lambda {I}_{\mathcal{H}}^{2}}} \}}},$

where λ is the smoothing parameter and n=Σ_(j=1) ^(N)n_(j) is the totalnumber of spikes. The solution to the variational problem can take theform

${\hat{I} = {\sum\limits_{j = 1}^{N}{\sum\limits_{k = 1}^{n_{j}}{c_{k}^{j}\varphi_{k}^{j}}}}},$

where c=[c₁ ¹,c₂ ¹, . . . , c_(n) ₁ ¹,c₁ ², c₂ ², . . . , c_(n) ₂ ², . .. , c_(n) _(N) ^(N)]^(T) satisfies the system of linear equationsG^(T)(G+nλI)c=G^(T)q with q=[q₁ ¹, q₂ ¹, . . . , q_(n) ₁ ¹, q₁ ², q₂ ²,. . . , q_(n) ₂ ², . . . ,q_(n) _(N) ^(N)]^(T), I the n×n identitymatrix and G the block matrix

${G = \begin{bmatrix}G^{11} & G^{12} & \ldots & G^{1N} \\G^{21} & G^{22} & \ldots & G^{2N} \\\vdots & \vdots & \ddots & \vdots \\G^{N\; 1} & G^{N\; 2} & \ldots & G^{N\; N}\end{bmatrix}},$

with entries of each block given by [G^(ij)]_(kl)=<φ_(k) ^(l),φ_(l)^(j)>. Thus, again the reconstruction problem reduces to solving asystem of linear equations.

In one embodiment, a recurrent neural network can be defined, similar tothe case of a single-input single-output TEM-encoded signal, by

${\frac{c}{t} = {{- \mu}{\nabla{E(c)}}}},$

with c=[c₁ ¹, c₂ ¹, . . . , c_(n) ₁ ¹, c₁ ², c₂ ², . . . , c_(n) ₂ ², .. . , c_(n) _(N) ^(N)]^(T), q=[q₁ ¹, q₂ ¹, . . . , q_(n) ₁ ¹, q₁ ², q₂², . . . , q_(n) ₂ ², . . . , q_(n) _(N) ^(N)]^(T), and

$G = \begin{bmatrix}G^{11} & G^{12} & \ldots & G^{1N} \\G^{21} & G^{22} & \ldots & G^{2N} \\\vdots & \vdots & \ddots & \vdots \\G^{N\; 1} & G^{N\; 2} & \ldots & G^{N\; N}\end{bmatrix}$

with entries of each block given by [G^(ij)]_(kl)=<φ_(k) ^(i), φ_(l)^(j)>.

In another embodiment, and again expanding the single-inputsingle-output TEM case, the reconstruction can be formed as a splineinterpolation problem

$\hat{I} = {\underset{{I\; {\varepsilon\mathcal{H}}},{\{{{\mathcal{L}_{}^{}I} = q_{k}^{j}}\}}_{{({k,j})} = {({1,1})}}^{({n,N})}}{argmin}{\{ {\mathcal{I}}_{\mathcal{H}}^{2\;} \}.}}$

The solution can take the form

${\hat{I} = {\sum\limits_{j = 1}^{N}{\sum\limits_{k = 1}^{n_{j}}{c_{k}^{j}\varphi_{k}^{j}}}}},$

and vector c is the solution to the optimization problem

$\begin{matrix}{{minimize}\mspace{14mu} \frac{1}{2}c^{T}G\; c} \\{{{{subject}\mspace{14mu} {to}\mspace{14mu} G\; c} = q},}\end{matrix}$

where c=[c₁ ¹, c₂ ¹, . . . , c_(n) ₁ ¹, c₁ ², c₂ ², . . . , c_(n) ₂ ², .. . , c_(n) _(N) ^(N)]^(T) satisfies the system of linear equationsG^(T)(G+nλI)c=G^(T)q with q=[q₁ ¹, q₂ ¹, . . . , q_(n) ₁ ¹, q₁ ², q₂ ²,. . . , q_(n) ₂ ², . . . , q_(n) _(N) ^(N)]^(T), I the n×n identitymatrix and G the block matrix

${G = \begin{bmatrix}G^{11} & G^{12} & \ldots & G^{1N} \\G^{21} & G^{22} & \ldots & G^{2N} \\\vdots & \vdots & \ddots & \vdots \\G^{N\; 1} & G^{N\; 2} & \ldots & G^{N\; N}\end{bmatrix}},$

with entries of each block given by [G^(ij)]_(kl)=<φ_(k) ^(i), φ_(l)^(j)>.

In one embodiment, a recurrent neural network can be defined, similar tothe case of a single-input single-output TEM-encoded signal, by

${{\frac{}{t}\begin{pmatrix}x \\y\end{pmatrix}} = {\beta \begin{pmatrix}{( {x - {\alpha \; Q\; x} + {\alpha \; A^{T}y}} )^{+} - x} \\{\alpha ( {{{- A}\; x} + q} )}\end{pmatrix}}},$

with c=[c₁ ¹, c₂ ¹, . . . , c_(n) ₁ ¹, c₁ ², c₂ ², . . . , c_(n) ₂ ², .. . , c_(n) _(N) ^(N)]^(T), q=[q₁ ¹, q₂ ¹, . . . , q_(n) ₁ ¹, q₁ ², q₂², . . . , q_(n) ₂ ², . . . , q_(n) _(N) ^(N)]^(T), and

$G = \begin{bmatrix}G^{11} & G^{12} & \ldots & G^{1N} \\G^{21} & G^{22} & \ldots & G^{2N} \\\vdots & \vdots & \ddots & \vdots \\G^{N\; 1} & G^{N\; 2} & \ldots & G^{N\; N}\end{bmatrix}$

with entries of each block given by [G^(ij)]_(kl)=<φ_(k) ^(i),φ_(l)^(j)>.

A schematic diagram of an exemplary embodiment of the disclosed subjectmatter is depicted in FIG. 4.

In some embodiments of the presently disclosed subject matter, the vTEMencoded signal can be encoded with neurons with a random threshold.Reconstruction of signals encoded with neurons with a random thresholdcan again be formulated as a variational approach, for example, byconsidering the reconstruction as the solution to a smoothing splineproblem. FIG. 5A-FIG. 5B is a schematic representation of a recurrentneural network for decoding signals encoded with a vTEM with neuronswith random threshold.

In one aspect of the disclosed subject matter, a system forreconstructing a TEM-encoded signal using a recurrent neural networkcomprises at least one input for receiving a TEM-encoded signal. Adders,integrators, multipliers, and/or piecewise linear activators can bearrange suitably such that the neural network is a map of at least onedifferential equation. The signal can be input into the at least oneinput, processed by the neural network, and output through at least oneoutput. The neural network can also have a feedback, such that the outerlayer sends a signal back to the first layer of the network. The outputscan be optionally integrated or otherwise processed.

In one embodiment, the system for reconstructing a TEM-encoded signalcan comprise a GPU cluster. For example, a GPU's intrinsically parallelarchitecture can be exploited to realize a recurrent neural network.Multiple GPUs can be used for signal reconstruction, hosts of which canbe connected using a switch fabric and peer-to-peer communicationaccomplished, for example, though the Message Passing Interface (MPI).

In another aspect of the disclosed subject matter, vTEM encoded signalscan be first divided into smaller volumes. The volumes can then bereconstructed and finally the reconstructed volumes can be stitchedtogether. For example, in one embodiment, a space-time video sequencecan be divided into fixed sized, overlapping volume segments. FIG. 6depicts an exemplary division of a space-time video sequence. Thestitching can be accomplished according to known methods. Exemplarystitching methods are disclosed in Lazar, A. A., Simonyi, E. K., & Toth,L. T., An overcomplete stitching algorithm for time decoding machines,IEEE Transactions on Circuits and Systems-I: Regular Papers, 55,2619-2630 (2008), the contents of which are incorporated by referenceherein.

The disclosed subject matter and methods can be implemented in softwarestored on computer readable storage media, such as a hard disk, flashdisk, magnetic tape, optical disk, network drive, or other computerreadable medium. The software can be performed by a processor capable ofreading the stored software and carrying out the instructions therein.

The foregoing merely illustrates the principles of the disclosed subjectmatter. Various modifications and alterations to the describedembodiments will be apparent to those skilled in the art in view of theteaching herein. It will thus be appreciated that those skilled in theart will be able to devise numerous techniques which, although notexplicitly described herein, embody the principles of the disclosedsubject matter and are thus within the spirit and scope of the disclosedsubject matter.

We claim:
 1. A method of reconstructing a signal encoded with a timeencoding machine (TEM) using a recurrent neural network, comprising: a)receiving a TEM-encoded signal; b) processing the TEM-encoded signal forinput into a recurrent neural network; c) reconstructing the TEM-encodedsignal with the recurrent neural network;
 2. The method of claim 1,wherein reconstructing further comprises a) formulating thereconstruction into a variational problem having a solution equal to theseries of sums of a sequence of functions multiplied by a sequence ofcoefficients, wherein the coefficients can be obtained by solving anoptimization problem; b) solving the optimization problem with arecurrent neural network, thereby generating the coefficients for thesolution; and c) reconstructing the signal with the coefficients.
 3. Themethod of claim 2, wherein the optimization problem is an unconstrainedoptimization problem with an explicit analytical solution.
 4. The methodof claim 3, wherein the recurrent neural network solves theunconstrained optimization problem using a general gradient approach. 5.The method of claim 4, wherein the recurrent neural network has aplurality of outputs, each output corresponding to a coefficient.
 6. Themethod of claim 1, wherein reconstructing further comprises a)formulating the reconstruction into a variational problem having asolution equal to the series of sums of a sequence of functionsmultiplied by a sequence of coefficients, wherein the coefficients canbe obtained by solving a spline interpolation problem; b) solving theinterpolation problem with a recurrent neural network, therebygenerating the coefficients to the solution; and c) reconstructing thesignal with the coefficients.
 7. The method of claim 6, wherein thespline interpolation problem is reformulated as a standard quadraticprogramming problem.
 8. The method of claim 1, wherein the TEM-encodedsignal is one dimensional, and wherein reconstructing further comprises:a) formulating the reconstruction as the variational problem${\hat{u} = {\underset{u \in \mathcal{H}}{argmin}\{ {{\sum\limits_{k = 1}^{n}\; ( {{\langle{u,\varphi_{k}}\rangle} - q_{k}} )^{2}} + {n\; \lambda {u}_{\mathcal{H}}^{2}}} \}}},$the solution of which takes the form${\hat{u} = {\sum\limits_{k = 1}^{n}\; {c_{k}\varphi_{k}}}},$ wherecoefficients c_(k) can be obtained by solving an unconstrainedoptimization problem minimize∥Gc−q∥_(1/2) ²+nλc^(T)Gc having an explicitanalytical solution G^(T)(G+nλI)c=G^(T)q, where c=[c₁, c₂, . . . ,c_(n)]^(T), q=[q₁, q₂, . . . q_(n)]^(T), and G is a symmetric matrixwith entries${\lbrack G\rbrack_{k,l} = {{\langle{\varphi_{k},\varphi_{l}}\rangle} = {\sum\limits_{m_{t} = {- M_{t}}}^{M_{t}}\; ( {\int_{k}^{k + 1}{( {D_{T}*e_{- m_{t}}} )(s)\ {{s} \cdot {\int_{l}^{l + 1}{( {D_{T}*e_{m_{t}}} )(s)\ {s}}}}}} )}}};$and b) solving the unconstrained optimization problem with a recurrentneural network.
 9. The method of claim 8, wherein a set of differentialequations, ${\frac{c}{t} = {{- \mu}\; {\nabla{E(c)}}}},$ withinitial condition c(0)=0, where E(c)=½(∥Gc−q∥_(1/2) ²+nλc^(T)Gc, andλ(c, t) is a n×n symmetric positive definite matrix that determines thespeed of convergence and whose entries are usually dependent on thevariables c(t) and time t, define the architecture of the recurrentneural network.
 10. The method of claim 9, wherein the recurrent neuralnetwork solves the unconstrained optimization problem using a generalgradient approach.
 11. The method of claim 1, wherein the TEM-encodedsignal is one dimensional, and wherein reconstructing further comprises:a) formulating the reconstruction as a spline interpolation problem,${\hat{u} = {\underset{{u \in \mathcal{H}},{\{{{\mathcal{L}_{\kappa}u} = q_{k}}\}}_{k = t}^{n}}{argmin}\{ {u}_{\mathcal{H}}^{2} \}}},$the solution to which takes the form${\hat{u} = {\sum\limits_{k = 1}^{n}\; {c_{k}\varphi_{k}}}},$ wherecoefficients c_(k) are the solution to the systems of linear equationsGc=q, and where coefficients c_(k) can be obtained by solving theoptimization problem $\begin{matrix}{{minimize}\frac{1}{2}c^{T}{Gc}} \\{{{subject}\mspace{14mu} {to}{\; \mspace{11mu}}{Gc}} = q}\end{matrix},$ where c=[c₁, c₂, . . . , c_(n)]^(T), q=[q₁, q₂, . . . ,q_(n)]^(T), and G is a symmetric matrix with entries${\lbrack G\rbrack_{k,l} = {{\langle{\varphi_{k},\varphi_{l}}\rangle} = {\sum\limits_{m_{t} = {- M_{t}}}^{M_{t}}\; ( {\int_{k}^{k + 1}{( {D_{T}*e_{- m_{t}}} )(s)\ {{s} \cdot {\int_{l}^{l + 1}{( {D_{T}*e_{m_{t}}} )(s)\ {s}}}}}} )}}};$b) solving the optimization problem with a recurrent neural network. 12.The method of claim 11, wherein the optimization problem is reformulatedas a standard quadratic programming problem, setting x=[x₊ ^(T)x⁻^(T)]^(T) and imposing x₊≧0, x⁻≧0 such that c=x₊−x⁻, giving a convexprogramming problem $\begin{matrix}{{minimize}\frac{1}{2}x^{T}{Qx}} \\{{{{subject}\mspace{14mu} {to}\mspace{14mu} {Ax}} = q},{x \geq 0}}\end{matrix},$ where ${Q = \begin{bmatrix}G & {- G} \\{- G} & G\end{bmatrix}},$ and where A=[G−G], and wherein the recurrent neuralnetwork is given by ${{\frac{}{t}\begin{pmatrix}x \\y\end{pmatrix}} = {\beta \begin{pmatrix}{( {x - {\alpha \; {Qx}} + {\alpha \; A^{T}y}} )^{+} - x} \\{\alpha ( {{- {Ax}} + q} )}\end{pmatrix}}},$ where (x)⁺=[(x₁)⁺, . . . , (x_(n))⁺]^(T) and(x₁)⁺=max{0,x_(i)}, α is a positive constant and β>0 is the scalingconstant.
 13. The method of claim 1, wherein the TEM-encoded signal is amultidimensional signal encoded with a video time encoding machine(vTEM).
 14. The method of claim 13, wherein reconstructing furthercomprises: a) formulating the reconstruction as the variational problem${\hat{I} = {\underset{l \in \mathcal{H}}{argmin}\{ {{\sum\limits_{j = 1}^{}\; {\sum\limits_{k = 1}^{n_{j}}\; ( {{\langle{I,\varphi_{k}^{j}}\rangle} - q_{k}^{j}} )^{2}}} + {n\; \lambda {I}_{\mathcal{H}}^{2}}} \}}},$the solution of which takes the form${\hat{I} = {\sum\limits_{j = 1}^{N}\; {\sum\limits_{k = 1}^{n_{j}}\; {c_{k}^{j}\varphi_{k}^{j}}}}},$where c=[c₁ ¹, c₂ ¹, . . . , c_(n) ₁ ¹, c₁ ², c₂ ², . . . , c_(n) ₂ ², .. . , c_(n) _(N) ^(N)]^(T) satisfies the system of linear equationsG^(T)(G+nλI)c=G^(T)q with q=[q₁ ¹, q₂ ¹, . . . , q_(n) ₁ ¹, q₁ ², q₂ ²,. . . , q_(n) ₂ ², . . . , q_(n) _(N) ^(N)]^(T), I the n×n identitymatrix and G the block matrix ${G = \begin{bmatrix}{G\;}^{11} & G^{12} & \cdots & G^{1N} \\G^{21} & G^{22} & \cdots & G^{2N} \\\vdots & \vdots & \ddots & \vdots \\G^{N\; 1} & G^{N\; 2} & \cdots & G^{NN}\end{bmatrix}},$ with entries of each block given by[G^(ij)]_(kl)=<φ_(k) ^(i), φ_(l) ^(j)>; b) solving the system of linearequations with a recurrent neural network
 15. The method of claim 14,wherein a set of differential equations,${\frac{c}{t} = {{- \mu}\; {\nabla{E(c)}}}},$ with initialcondition c(0)=0, where E(c)=½(∥Gc−q∥_(1/2) ²+nλc^(T)Gc), and μ(c, t) isa n×n symmetric positive definite matrix that determines the speed ofconvergence and whose entries are usually dependent on the variablesc(t) and time t, define the architecture of the recurrent neuralnetwork.
 16. A system for reconstructing a signal encoded with a timeencoding machine (TEM) using a recurrent neural network circuit,comprising: a) at least one input for receiving a TEM-encoded signal,wherein the TEM-encoded signal comprises at least one spike train; b) atleast one processor electrically coupled with the at least one input fordetermining characteristics of the at least one spike; c) a recurrentneural network circuit electrically coupled with the at least oneprocessor, wherein the recurrent neural network circuit is configuredwith reference to the characteristics of the at least one spike train,and wherein the recurrent neural network circuit comprises anarrangement of adders, integrators, multipliers, and/or piecewise linearactivators; and d) at least one output electrically coupled to therecurrent neural network circuit for providing a reconstructed signal.17. The system of claim 16, wherein the recurrent neural network circuitis arranged into three layers, comprising: a) a first layer, consistingof a plurality (n) of multiply/add units, wherein the vector (G+nλI)c−qis computed, wherein the multiplication factors are entries of thematrix G+nλI and the vector q; b) a second layer, consisting of aplurality (n) of multiply/add units, wherein ∇E(c) is evaluated, where∇E(c)=G^(T)((G+nλI)c−q), and wherein multiplication factors are providedby the entries of matrix G; c) a third layer, consisting of a plurality(n) of multiply/add units, wherein the gradient is weighted by thelearning rate μ, and wherein the outputs of the third layer provide thetime derivative of the vector c(t), the outputs of the third layer beingintegrated and fed back into the first layer.
 18. The system of claim16, wherein the recurrent neural network is arranged into a singlelayer, comprising: a) a plurality (n) of multiply/add units, wherein avector (q=Gc) is computed, and wherein a plurality of multiplicationfactors are entries of a matrix G and the vector q; b) a plurality ofoutputs, wherein the outputs provide a time derivative of a vector c(t),the outputs being integrated and fed back into the first layer.
 19. Thesystem of claim 16, wherein the at least one input comprises a singleinput, and the at least one output comprises a plurality of outputs. 20.The system of claim 16, wherein the at least one input comprises aplurality of inputs, and the at least one output comprises a pluralityof outputs.